This application is based on Japanese Patent Application No. 9-133159 filed on May 23, 1997, the contents of which is incorporated herein by reference.
1. Field of the Invention
This invention relates to a system for displaying combined imagery, and more particularly to a system for displaying virtual imagery blended with real images with the aid of depth information in an augmented reality (AR) system.
2. Description of the Prior Art
A system for producing virtual reality (VR) has been devised, which is concerned with letting one feel as if computer-generated visuals were the real world. The VR system allows one to experience every detail of the realities of life in a virtual environment. The VR is also called artificial reality.
The aforesaid AR system is a kind of the VR system developed for enhancing the user's vision with computer-generated imagery.
Although the VR system allows a user to be completely immersed in the virtual environment, the AR system aims at improving the quality of a real environment by properly disposing the virtual environment therein. By the expression "AR system" as used herein is meant a visual AR system.
Various cases which are exemplary of the utility of the AR system can be enumerated. For example, an image obtained from a computerized tomographic scanner is overlaid on the real brain of a patient being operated by brain surgeons; the work of assembling or fixing up a machine is supported by notes appended to the real parts by computer graphics (CG); a person who is thinking of doing over his or her room is allowed beforehand to judge of the arrangement of new furniture by its effect upon his or her mind; or persons in charge of city redevelopment are allowed to have virtual sight of a full-sized row of stores and houses at the site of construction before they are actually built.
See-through head-mounted displays (STHMD's) to be used for the AR system are available in two types, optical and video. FIGS. 8 and 9 are diagrams for the principles of operation of the AR systems 80 and 90 provided with optical and video STHMD's respectively in accordance with the prior art.
The AR system 80 shown in FIG. 8 includes a half mirror 81 disposed on the visual axis of an observer so as to afford to him or her an overall view of the real object field. Virtual imagery are cast on the half mirror 81 by a projector 82 and reflected to the observer's eyes. Consequently, the real object field is seen through the virtual imagery. A head position detector 83 mounted on the observer's head transmits a signal to a workstation 84 when the former detects a change in the observer's visual point caused by his or her carriage of looking aside, upward or downward. Then the workstation 84 transmits a signal to the projector 82 so as to change virtual imagery in accordance with a change in the observer's visual point.
The AR system 90 shown in FIG. 9 includes two cameras 93 disposed in conjugate relationship with the observer's eyes respectively and adapted to photograph the real object field which would be seen to the observer if it were not for a double-faced mirror 91. A workstation 95 receives real images from the cameras 93 and combines them with virtual imagery. Combined imagery resulting therefrom are cast on the double-faced mirror 91 by the projector 82 and reflected to the observer's eyes.
It has been pointed out that the AR system requires complete registration between imagery which should appear in the foreground of a final picture and imagery which should appear in the background and that the portion of the latter on which the former is overlaid should be hidden behind the former. The second mentioned requirement is not fulfilled by either of the aforesaid AR systems 80 and 90.
More specifically stated, the AR system 80 is necessarily foredoomed to allowing the real object field to be seen through the virtual imagery. Thus the virtual imagery is merely superimposed over the real object field.
The AR system 90 is such that the virtual imagery always appear in the foreground of the final picture and, consequently, the portion of the real images on which the virtual imagery are overlaid is hidden and left invisible behind the virtual imagery. This is the case even when the real images have to appear in the foreground of the final picture.
The recent progress in the psychology of perception has revealed that improper or unexpected stereoscopic vision is caused by a wrong positional relationship between the real object field and virtual imagery, because such a wrong positional relationship is contradictory to the information on the actual vergence and parallax. This finding provides a conclusion that a correct positional relationship between the real object field and virtual imagery is of prime importance in the AR system.
A previously proposed apparatus of this type described in U.S. patent application Ser. No. 08/896,973 is based on the fact that, when an image which should appear in the foreground of a final picture is overlapped on an image which should appear in the background of the final picture, pixels representing one of the overlapped portions of these two images are located at the same locations as pixels representing the other thereof. The distance between the apparatus and one image is compared with the distance between the apparatus and the other image. The pixels which represent the image disposed closer to the observer are selected for the purpose of display.
Thus the above-described previously proposed apparatus requires depth information with respect to each pixel representing a real image. A method of obtaining this information has heretofore been proposed where two-dimensional images are obtained from two cameras, respectively, arranged along a line perpendicular to the optical axes of the cameras. By means of correlation analysis, a point which corresponds to a point on one of the two-dimensional images is found on the other thereof. The aforesaid information is obtained by applying a trigonometric relation to these two points.
The above-described method has the disadvantages that corresponding points cannot be detected from the flat portions of real images and that wrong points are mistaken for corresponding points in the presence of patterns similar to each other.
A light section method, in which an object is actively irradiated by light beams, may solve the aforesaid problems. In this case, however, a sensor for receiving the light beams reflected on the surface of the object has to be provided separately from the cameras, and this necessitates a means for allowing the visual point of the sensor to agree with that of the cameras. If the visual points do not agree with each other, a measured distance differing from the actual distance will hinder the aforesaid second mentioned requirement from being fulfilled.